Skip to content

Phase B3b — zero-copy persisted FM-index - #16

Merged
logannye merged 12 commits into
mainfrom
rosalind/phase-b3b-zerocopy-index
May 27, 2026
Merged

Phase B3b — zero-copy persisted FM-index#16
logannye merged 12 commits into
mainfrom
rosalind/phase-b3b-zerocopy-index

Conversation

@logannye

Copy link
Copy Markdown
Owner

Summary

The keystone of Phase B's persistence pillar: build the reference index once into a portable artifact; thereafter every analysis memory-maps it (never rebuilds), runs in bounded memory, and is byte-identically reproducible — delivered at the library level.

  • Persists a built GenomeIndex to a deterministic, versioned, 8-aligned little-endian on-disk artifact (IndexWriter::write_genome_index) and loads it as a zero-copy, memory-mapped FmIndexView that answers backward_search / sa_at / locate_exact in place — no rebuild, no allocation of the bulk index.
  • Architecture: one FM algorithm, two backings. The query algorithm lives exactly once as generic free functions over a pub(crate) BwtBacking trait (src/genomics/fm_backing.rs); both the in-RAM BlockedFMIndex and the borrowed FmIndexView implement it, so the view cannot diverge on logic — only on how it fetches bytes. A byte-identical equivalence gate proves it.
  • IndexReader::open fully validates the artifact: magic/version/endian, every section's extent + 8-alignment, contig global-offset consistency, per-block-record extents + 8-alignment, and SA-sample-rate consistency. Corrupt/truncated/misaligned files are rejected at open — never mis-queried or crashed-on at query time.
  • Dependency-free zero-copy via checked slice::align_to (little-endian host; no bytemuck). docs/index-format.md documents the format as a forkable contract, including the faithful-serialization size note and the v2 lean-rank path.

Scope: library-level (B3b). Deferred by design: rosalind index CLI + load wiring (B3c); wiring align/variants/pileup onto the persisted index (B4); MemoryBudget enforcement (Phase C); leaner/faster rank (format v2).

Spec gates — all met

  • Equivalence: view == in-RAM, byte-identical over a multi-contig + N-bearing + boundary-straddle battery (exhaustive rank/symbol_at over every position, plus backward_search/sa_at/locate_exact).
  • Determinism: byte-identical across two independent builds of the same reference.
  • No rebuild on load: the read path never calls SA-IS / BlockedFMIndex::build (structurally grepped + behaviorally proven by a self-contained query after dropping the in-RAM index).
  • Bounded residency: FmIndexView is a small borrow (slices + scalars), not an owned copy; size independent of genome size.
  • Behavior preserved: the B3b.1 BwtBacking refactor changes no observable behavior (existing FM-index/aligner/genome_index suites + fm_index_props are the witnesses).

Test plan

  • cargo test — full suite green (23 test binaries, 0 failures)
  • cargo build0 warnings
  • cargo fmt --all -- --check — clean
  • New tests/index_persistence.rs (public-API equivalence battery, determinism, self-contained query, bounded residency) + the view == in-RAM equivalence gate + corruption-rejection unit tests (truncated / bad-magic / corrupt block directory / misaligned block offset)
  • MSRV 1.72 preserved (no div_ceil); two unsafe align_to blocks reviewed and found sound (checked prefix/suffix + LE-host guard + per-record validation at open)

Design spec: docs/superpowers/specs/2026-05-27-phase-b3b-zerocopy-index-design.md. Plan: docs/superpowers/plans/2026-05-27-phase-b3b-zerocopy-index.md.

🤖 Generated with Claude Code

logannye and others added 12 commits May 27, 2026 10:27
Capability-anchored (build-once -> instant-start, bounded, reproducible). BwtBacking seam (one algorithm, owned + borrowed backings); deterministic 8-aligned LE on-disk format extending genomics/index/format.rs; dependency-free zero-copy via align_to; faithful serialization in v1 (lean-rank is a versioned v2); equivalence + determinism + no-rebuild gates. Decomposed B3b.1 (trait) -> B3b.2 (format+view+gate+docs/index-format.md).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8-task plan across two parts: B3b.1 (BwtBacking trait + generic FM ops, route BlockedFMIndex through it, behavior unchanged) and B3b.2 (8-aligned LE on-disk format + deterministic GenomeIndex serializer + zero-copy FmIndexView/GenomeIndexView + view==in-RAM equivalence/determinism/no-rebuild/bounded-residency gates + docs/index-format.md). Derived from the committed spec (4c55b8a) and grounded in the existing fm_index/rank_select/sampled_sa/index code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracts the data surface the blocked FM query algorithm reads into a pub(crate) BwtBacking trait, and rewrites rank/symbol_at/lf_index/backward_search/sa_at/locate_interval/total as free functions generic over it. The single source of truth for the algorithm; the owned BlockedFMIndex (next) and the persisted FmIndexView (B3b.2) both run it, so the borrowed view can only diverge on backing, not logic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BlockedFMIndex implements BwtBacking over its owned fields; backward_search/locate_interval/rank/total/symbol_at/sa_at become thin wrappers over the generic fm_backing ops, and the private lf_index is removed (now fm_backing::lf_index). No observable behavior change — the FM-index/aligner/genome_index suites and tests/fm_index_props are the witnesses. The inherent c_table() -> &[u32;6] is kept for align_within_block; the trait's by-value c_table coexists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…itives

Adds additive pub(crate) accessors so the B3b serializer can read each structure's backing words (BWTBlock::{start,end,bwt,occ,sentinel_offset}, RankSelectIndex::{bitvectors,superblocks,len}, SampledSuffixArray::{marks,superblocks,values}, CompressedBoundaries::len), and promotes popcount_range + RANK_STRIDE to pub(crate) so the borrowed FmIndexView computes rank with the same code as the owned structures. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ions

Adds Reference2bit/FmMeta/Boundaries/Blocks section kinds (the B3b on-disk format extends the existing versioned v1 header; Reference stays reserved). The fixed header is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
IndexWriter::write_genome_index serialises a built GenomeIndex into the 8-aligned little-endian section format (Contigs+global_offset, Reference2bit, FmMeta, Boundaries, self-describing Blocks directory+records, SaSamples). Byte-identical across repeated builds. The scaffold write_v1/ContigInfo and the old reader are removed; IndexReader::open is stubbed pending Task 6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
open() mmaps the file, validates the header + every section extent/8-alignment, parses the new Contigs layout into a ContigSet (recomputing and integrity-checking global offsets), and locates the FM sections. Rejects truncated/bad-magic files. The borrowed FmIndexView (Task 7) interprets the FM sections.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…equivalence gate

FmIndexView reads the FM sections as &[u64]/&[u32] over the mmap (checked align_to, little-endian host) and implements BwtBacking, so backward_search/sa_at/locate_interval run the same generic ops as the in-RAM index. GenomeIndexView pairs it with the ContigSet for locate_exact. The equivalence gate asserts byte-identical backward_search/sa_at/locate_exact/rank/symbol_at over a multi-contig + N-bearing battery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per code review: FmIndexView::new now validates every block record (directory offset + declared word-counts) fits within the Blocks section using checked arithmetic, so a corrupt-but-open-valid file is rejected at open() rather than panicking or silently wrapping into a wrong-length slice at query time (release builds have no overflow checks). Also asserts the SaSamples rate equals the FmMeta sa_sample_rate (removing the previously-unused SampledView.rate field). Adds open_rejects_a_corrupt_block_directory.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Integration gates over the public API: view == in-RAM locate_exact over a multi-contig + N battery, byte-identical builds (two independent builds), self-contained query (in-RAM index dropped), and bounded residency (FmIndexView is a small borrow, not an owned copy). Documents the versioned on-disk format as a forkable contract incl. integrity validation and the v2 lean-rank path. Removes the unused RankSelectIndex::len accessor for a clean build.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final-review fix: validate_block_records now rejects a block-record directory offset that is in-bounds but not 8-aligned, so a corrupt index is rejected at open() rather than panicking at query time via block()'s align_to .expect (the integrity contract: corrupt is rejected up front, never crashed-on later). Adds open_rejects_a_misaligned_block_offset. Also: drop a useless vec! in the header reserve, and document the MSRV-mandated (n+7)/8*8 (not div_ceil) at the two block-size sites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@logannye
logannye merged commit 990ae07 into main May 27, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant